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ABSTRACT 

The  FPJE  was  an  experiment  to  consider  the  best  way  to  develop  and  evaluate  a  system  of  systems  approach  to  Force 
Protection.  It  was  sponsored  by  Physical  Security  Equipment  Action  Group  (PSEAG)  and  Joint  Program  Manager  - 
Guardian  (JPM-G),  and  was  managed  by  the  Product  Manager  -  Force  Protection  Systems  (PM-FPS).  The  experiment 
was  an  effort  to  utilize  existing  technical  solutions  from  all  branches  of  the  military  in  order  to  provide  more  efficient 
and  effective  force  protection.  The  FPJE  consisted  of  four  separate  Integration  Assessments  (lA),  which  were  intended 
as  opportunities  to  assess  the  status  of  integration,  automation  and  fusion  efforts,  and  the  effectiveness  of  the  current 
configuration  and  “system”  components.  The  underlying  goal  of  the  FPJE  was  to  increase  integration,  automation,  and 
fusion  of  the  many  different  sensors  and  their  data  to  provide  enhanced  situational  awareness  and  a  common  operational 
picture. 

One  such  sensor  system  is  the  Battlefield  Anti-Intrusion  System  (BAIS),  which  is  a  system  of  seismic  and  acoustic 
unmanned  ground  sensors.  These  sensors  were  originally  designed  for  employment  by  infantry  soldiers  at  the  platoon 
level  to  provide  early  warning  of  personnel  and  vehicle  intrusion  in  austere  environments.  However,  when  employed 
around  airfields  and  high  traffic  areas,  the  sensitivity  of  these  sensors  can  cause  an  excessive  number  of  detections. 
During  the  second  FPJE-IA  all  of  the  BAIS  detections  and  the  locations  of  all  Opposing  Forces  were  logged  and 
analyzed  to  determine  the  accuracy  rate  of  the  sensors.  This  analysis  revealed  that  with  minimal  filtering  of  detections, 
the  number  of  false  positives  and  false  negatives  could  be  reduced  substantially  to  manageable  levels  while  using  the 
sensors  within  extreme  operational  acoustic  and  seismic  noise  conditions  that  are  beyond  the  design  requirements. 
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1.  BACKGROUND 

The  Battlefield  Anti-Intrusion  System  (BAIS)  consists  of  a  hand  held  monitor  and  one  or  more  sensors  units.  The  basic 
sensor  unit  has  both  seismic  and  acoustic  sensors,  but  other  modules  including  passive  infrared,  and  magnetic  sensors  are 
also  available.  These  additional  modules  provide  higher  probability  of  detection  and  much  lower  false  alarm  rates  so 
they  are  not  included  in  this  analysis. 

The  BAIS  theory  of  operation  is  based  on  a  sampling  of  the  seismic  and  acoustic  sensors  over  a  rolling  10  second  time 
window.  If  over  that  10  second  interval,  one  or  more  events  was  detected  that  are  above  the  threshold  value,  then  a 
classification  algorithm  will  be  applied  and  the  results  reported  to  the  hand  held  receiver  module  via  RF.  The 
classification  algorithm  groups  events  into  the  following  categories:  personnel,  vehicle,  wheeled  vehicle,  tracked  vehicle, 
or  unknown.  For  the  purpose  of  analysis,  it’s  important  to  recognize  that  the  events  reported  to  the  hand  held  receiver  do 
not  provide  a  true  count  of  events,  but  a  list  of  time  slices  in  which  one  or  more  events  may  have  occurred. 

The  purpose  of  this  document  is  to  illustrate  the  ineffectiveness  of  the  baseline  seismic/acoustic  BAIS  sensors  in  high 
traffic  areas  during  the  Force  Protection  Joint  Experiment  (FPJE)  without  any  filtering  and  to  show  how  the 
effectiveness  was  greatly  improved  with  basic  filtering.  There  are  many  ways  to  filter  sensor  data,  and  this  document 
only  addresses  a  few  of  the  initial  approaches  that  were  used  during  the  FPJE.  The  intent  was  to  develop  a  heuristic 
algorithm  based  on  comparing  sensor  detections  with  true  data.  These  sensors  only  report  the  result  of  their  own  internal 
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classification  algorithm,  precluding  the  use  a  more  complex  mathematical  filter  that  could  use  the  raw  amplitude  and 
frequency  measurements  to  further  filter  out  false  detections. 


2.  ANALYSIS  TERMINOLOGY 


2.1  Baseline 

When  there  is  no  filtering  of  the  data  for  a  given  analysis.  The  baseline  for  the  analysis  of  all  detection  types  will  differ 
from  the  baseline  for  the  analysis  of  just  personnel  detections. 

2.2  Fusion-Engine 

When  a  sensor  creates  a  detection,  it  is  run  through  the  fusion-engine  to  determine  whether  to  allow  the  detection  to  pass 
through  or  to  ignore  it  as  noise. 

2.3  False-Positive 

When  a  sensor  creates  a  detection,  or  the  fusion-engine  allows  a  detection  to  pass  through,  but  there  is  no  known  OPFOR 
(Opposing  Force)  within  a  predetermined  distance  from  the  sensor.  In  other  words  there  is  a  detection  but  there  should 
not  have  been.  We  check  the  distance  of  the  OPFOR  from  the  sensor  for  each  detection  reported  by  the  fusion-engine, 
not  just  the  current  detection. 

2.4  False-Negative 

When  there  is  a  known  OPFOR  within  a  predetermined  distance  from  a  sensor,  but  that  sensor  does  not  create  a 
detection,  or  the  fusion-engine  does  not  allow  a  detection  to  pass  through.  In  other  words  there  is  no  detection  but  there 
should  have  been.  However  there  is  a  special  case  where  the  fusion-engine  does  not  allow  a  detection  through,  but  then 
later  utilizes  that  detection  in  its  decision  to  allow  a  detection  through.  In  this  case,  we  claim  that  the  first  detection  was 
not  a  false-negative. 

2.5  True-Positive 

When  a  sensor  creates  a  detection,  or  the  fusion-engine  allows  a  detection  to  pass  through,  and  there  is  a  known  OPFOR 
within  a  predetermined  distance  from  the  sensor.  In  other  words  there  is  a  detection  and  there  should  have  been. 

2.6  True-Negative 

When  a  sensor  does  not  create  a  detection,  or  the  fusion-engine  does  not  allow  a  detection  to  pass  through,  and  there  was 
no  known  OPFOR  within  a  predetermined  distance  from  the  sensor.  In  other  words  there  is  no  detection  and  there  should 
not  have  been. 


3.  ANALYSIS 


3.1  Filters  in  the  Fusion-Engine 

The  fusion-engine  for  the  purpose  of  this  analysis  is  given  a  detection  threshold,  a  time  period  in  seconds,  and  a 
classification.  It  then  searches  through  previous  detections  up  the  given  number  of  seconds  prior  to  the  current  detection. 
It  counts  the  number  of  detections  of  the  given  classification  that  occur  on  the  same  sensor  in  the  given  time  period.  If 
the  count  is  greater  than  or  equal  to  the  given  detection  threshold,  then  the  fusion-engine  allows  the  detection  to  pass 
through.  Note  that  the  highest  time  period  considered  during  initial  analysis  for  a  classification  was  60  seconds. 

3.2  Timing  Considerations 

The  BAIS  sensor  sends  detections  to  the  BAIS  receiver  at  most  once  every  ten  seconds.  It  is  important  to  note  that  while 
there  is  a  limit  of  once  every  ten  seconds  at  the  receiver;  there  is  no  such  limit  once  the  data  reaches  JBC2S.  The  results 
are  based  on  this  limiting  factor,  but  because  the  timestamps  are  set  when  JBC2S  logs  them,  there  are  situations  where 
the  process  can  slow  down  and  two  detections  can  be  logged  within  ten  seconds. 


3.3  All  Detections 

There  are  five  different  classifications  that  the  sensors  give  to  detections:  personnel,  vehicle,  wheeled  vehicle,  tracked 
vehicle,  and  unknown.  This  section  analyzes  the  sensors  no  matter  what  classification  was  given.  The  predetermined 
distance  that  is  acceptable  for  detections  is  350  meters  based  on  the  BAIS  manual. 

In  Figure  1  you  will  see  the  baseline  case  and  the  filtering  cases  where  the  detection  threshold  was  two  and  three 
detections.  A  threshold  of  two  means  that  there  were  two  detections  in  the  given  time  period,  and  the  figure  shows  the 
confidence  in  the  detections.  There  a  few  points  to  note  in  this  figure.  First,  notice  that  85.4%  of  the  detections  in  the 
baseline  are  false-positives,  which  is  quite  high.  However,  with  the  filtering  shown,  the  number  of false-positives 
reduces  greatly,  exactly  how  much  depending  on  the  filtering  variables.  Now,  notice  that  there  are  no  fals e-negatives  for 
the  baseline  and  yet  there  are  false-negatives  for  the  filtered  cases.  This  shows  that  the  while  the  filters  manage  to  reduce 
the  number  of false-positives  they  unfortunately  add  false-negatives. 


All  -  350M 


■  False  Postives  ■  False  Negatives  BTrue  Pos  &  Neg 
Figure  1 :  All  Detections  with  Threshold  at  Two  and  Three 


There  are  the  same  interesting  features  in  Figure  2,  which  shows  the  baseline  again  and  the  filtering  cases  where  the 
detection  threshold  was  four,  five,  six,  and  seven.  Another  feature  of  the  data  that  becomes  apparent  in  this  figure  is  that 
there  seems  to  be  less  data  for  each  scenario  as  the  detection  threshold  increases.  Specifically  notice  that  there  is  no  data 
for  four  detections  in  ten  seconds  or  five  in  ten  or  five  in  20.  These  scenarios  were  not  ignored;  they  were  simply  not 
included  in  the  chart  because  they  created  a  situation  where  we  all  detections  were  ignored,  because  the  scenario  never 
occurred.  In  other  words,  there  never  were  four  detections  in  ten  seconds,  so  the  scenario  gave  us  useless  data. 


All  -  350M 
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Figure  2:  All  Detections  with  Threshold  at  Four  through  Seven 


The  scenario  where  the  detection  threshold  is  two  shows  that  as  the  time  increases  for  a  given  scenario,  the  false- 
positives  and  the  fals e-negatives  respond  inversely  with  the  fals e-positives  increasing  and  the  fals e-negatives  decreasing. 
We  will  see  that  this  is  a  common  theme  throughout  the  analysis.  Without  any  filtering  we  have  the  baseline  case  which 
is  the  one  extreme  of  this  inverse  relationship,  with  a  very  high  number  of false-positives  and  zero  fals  e-negatives,  which 
is  also  a  theme  of  the  baselines.  So  for  each  detection  threshold,  the  goal  is  to  show  the  other  extreme  of  very  low  false- 
positives  and  relatively  higher  false-negatives. 

In  this  next  section,  there  is  a  chart  for  detection  thresholds  two  and  three.  Both  charts  focus  in  on  the  time  period  that 
causes  the  number  of false-positives  to  drop  close  to  zero.  There  is  never  a  detection  threshold  and  time  period 
combination  that  will  have  zero  false-positives  on  any  of  these  charts.  This  is  because  in  the  cases  where  there  are  zero 
false-positives,  it  is  a  result  of  filtering  out  all  detections  which  by  default  eliminates  all  false-positives.  However,  this  is 
not  a  very  good  filtering  solution,  so  these  cases  are  not  displayed. 
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Figure  3:  All  Detections  with  Threshold  at  Two 
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Figure  4:  All  Detections  with  Threshold  at  Three 


In  some  cases  like  in  Figure  4  we  jump  from  ten  seconds  to  14  seconds  along  the  horizontal  axis.  This  is  because  there  is 
no  change  from  ten  to  14  seconds,  so  we  only  show  the  edges  of  the  time  gaps  with  the  same  results. 


The  main  thing  to  notice  from  these  two  charts  is  that  the  more  detections  you  wait  for,  the  more  confidence  there  is  in  a 
true-positive  or  true-negative.  Also,  on  both  charts,  the  change  from  one  second  to  another  is  only  a  few  false-positives, 
which  given  a  total  of  2500  detections,  is  only  a  fraction  of  a  percentage  in  change.  Considering  the  baseline  is  85.4%, 
any  reduction  in  false-positives  below  40%  is  a  substantial  improvement. 


3.4  Personnel  Detections 

This  section  analyzes  all  detection  of  classification  type  personnel.  This  is  a  very  small  subsection  of  the  total  detections, 
with  only  186  detections.  The  predetermined  distance  that  is  acceptable  for  personnel  detections  is  50  meters  based  on 
the  BAIS  manual. 


In  Figure  5  you  will  see  the  baseline  case  and  the  filtering  cases  where  the  detection  threshold  was  two  and  three 
detections.  Similar  to  the  section  with  all  classifications,  the  baseline  for  personnel  has  a  high  number  of false-positives 
with  77.4%.  Again  we  see  that  the  filtering  reduces  the  number  of false-positives,  but  creates  some  false-negatives.  In 
Figure  10  you  see  the  baseline  repeated  and  the  filtering  cases  where  the  detection  threshold  was  four,  five,  six,  and 
seven. 


Figures  7  and  8  are  charts  for  detection  thresholds  two  and  three.  Both  charts  focus  in  on  the  time  period  that  causes  the 
number  of false-positives  to  drop  close  to  zero.  The  main  thing  to  notice  on  all  the  charts  in  this  section  is  that  personnel 
detections  are  relatively  reliable.  Even  with  a  small  detection  threshold  of  two,  like  in  Figure  7,  the  number  of false- 
positives  is  low.  As  you  increase  the  detection  threshold,  the  false-positives  remain  low  and  the  false-negatives  decrease. 


Figure  7:  Personnel  Deteetions  with  Threshold  at  Two  Figure  8:  Personnel  Deteetions  with  Threshold  at  Three 
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3.5  Vehicle  Detections 

This  section  analyzes  all  detection  of  classification  type  vehicle.  This  is  a  fairly  large  subsection  of  the  total  detections, 
with  1,262  detections.  The  predetermined  distance  that  is  acceptable  for  vehicle  detections  is  350  meters  based  on  the 
BAIS  manual. 

In  Figure  9  you  will  see  the  baseline  case  and  the  filtering  cases  where  the  detection  threshold  was  two  and  three 
detections.  Again,  the  baseline  for  vehicle  has  a  high  number  of false-positives  with  87.7%.  Again  we  see  that  the 
filtering  reduces  the  number  of false-positives,  but  creates  some  fals e-negatives.  In  Figure  10  you  see  the  baseline 
repeated  and  the  filtering  cases  where  the  detection  threshold  was  four,  five,  and  six.  Unlike  for  all  detections  and  for 
personnel  detections,  the  highest  detection  threshold  shown  is  six.  This  is  because  for  higher  thresholds,  the  fusion- 
engine  would  never  allow  any  detections  through.  For  example,  let’s  say  the  threshold  is  seven  detections.  The  fusion- 
engine  would  never  let  a  detection  through,  because  there  was  never  seven  detections  within  the  given  time  periods,  the 
max  of  which  was  60  seconds. 
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Figure  9:  Vehicle  Detections  with  Threshold  at  Two  and  Three 


Figures  1 1  and  12  are  charts  for  detection  thresholds  two  and  three.  Both  charts  focus  in  on  the  time  period  that  causes 
the  number  of false-positives  to  drop  close  to  zero.  The  main  thing  to  notice  from  these  charts  is  that  as  you  increase  the 
detection  threshold,  the  false-positives  remain  low  and  the  fals  e-negatives  decrease.  Also,  as  the  threshold  increase  over 
three,  there  is  not  much  added  confidence. 
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Figure  11:  Vehicle  Detections  with  Threshold  at  Two 
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Figure  12:  Vehicle  Detections  with  Threshold  at  Three 


3.6  Wheeled  Vehicle  Detections 

This  section  analyzes  all  detection  of  classification  type  wheeled  vehicle.  There  are  856  detections  in  this  sample  of  the 
total  detections.  The  predetermined  distance  that  is  acceptable  for  wheeled  vehicle  detections  is  250  meters  based  on  the 
BAIS  manual. 


In  Figure  13  you  will  see  the  baseline  case  and  the  filtering  cases  where  the  detection  threshold  was  two  and  three 
detections.  Again,  the  baseline  for  wheeled  vehicle  has  a  high  number  of false-positives  with  94.0%.  Again  we  see  that 
the  filtering  reduces  the  number  of false-positives,  but  creates  some  false-negatives.  In  Figure  14  you  see  the  baseline 
repeated  and  the  filtering  cases  where  the  detection  threshold  was  four,  five,  six,  and  seven. 

Figures  15  and  16  are  charts  for  detection  thresholds  two  and  three.  Both  charts  focus  in  on  the  time  period  that  causes 
the  number  of false-positives  to  drop  close  to  zero.  The  main  thing  to  notice  from  these  charts  is  that,  again,  as  the 
threshold  amount  increases  above  three  it  has  little  to  no  effect  on  the  number  of false-positives. 


SPIE  Multisensor,  Multisource  Information  Fusion,  Orlando,  FL,  13-17  April,  2009 


Wheeled  Vehicle  - 
250M 


Wheeled  Vehicle  - 
250M 


■  False  Postives  ■  False  Negatives  BTrue  Pos  &  Neg 


■  False  Postives  ■  False  Negatives  BTrue  Pos  &  Neg 


Figure  15:  Wheeled  Vehiele  Deteetions  with 
Threshold  at  Two 


Figure  16:  Wheeled  Vehiele  Deteetions  with 
Threshold  at  Three 


3.7  Tracked  Vehicle  Detections 

This  section  analyzes  all  detection  of  classification  type  tracked  vehicle.  This  is  an  extremely  small  subsection  of  the 
total,  with  only  13  detections.  The  predetermined  distance  that  is  acceptable  for  tracked  vehicle  detections  is  350  meters 
based  on  the  BAIS  manual.  This  low  number  of  detections  for  tracked  vehicles  makes  sense  because  the  OPFOR  was  not 
traveling  on  tracked  vehicles. 


In  Figure  17  you  will  see  the  baseline  case  and  the  filtering  cases  where  the  detection  threshold  was  two,  three,  and  four 
detections.  Despite  the  extremely  low  number  of  detections  in  this  sample,  you  see  the  same  pattern  as  seen  with  other 
samples.  In  the  baseline  we  see  84.6%  false-positive  rate.  Because  of  the  low  number  of  detections  in  this  sample,  all  we 
can  really  note  from  this  chart  is  that  the  pattern  is  consistent  even  with  a  small  sample. 
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Figure  17:  Tracked  Vehicle  Detections  with  Threshold  at  Two  through  Four 


3.8  Unknown  Detections 

This  section  analyzes  all  detection  of  classification  type  unknown.  This  is  a  small  subsection  of  the  total,  with  only  183 
detections.  The  predetermined  distance  that  is  acceptable  for  unknown  detections  is  350  meters  based  on  the  BAIS 
manual. 

In  Figure  18  you  will  see  the  baseline  case  and  the  filtering  cases  where  the  detection  threshold  was  two,  three,  four,  and 
five  detections.  Again,  the  baseline  for  unknown  has  a  high  number  of false-positives  with  84.2%.  Again  we  see  that  the 
filtering  reduces  the  number  oi  false-positives,  but  creates  somQ  false-negatives. _ 

Unknown  -  350M 
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Figure  18:  Unknown  Detections  with  Threshold  at  Two  through  Five 

Figures  19  and  20  are  charts  for  detection  thresholds  two  and  three.  Both  charts  focus  in  on  the  time  period  that  causes 
the  number  of false-positives  to  drop  close  to  zero.  The  main  thing  to  notice  from  these  charts  is  that  for  the  higher 
thresholds,  a  relatively  small  change  in  acceptable  time  has  little  to  no  effect  on  the  number  of false-positives.  Also,  after 
a  threshold  of  three,  there  is  not  much  improvement  in  results. 
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Figure  19:  Unknown  Detections  with 
Threshold  at  Two 
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I  False  Postives  ■  False  Negatives  BTrue  Pos  &  Neg 

Figure  20:  Unknown  Detections  with 
Threshold  at  Three 


3.9  All  Vehicle  Detections 

This  section  analyzes  all  detections  of  classification  types,  including  vehicle,  wheeled  vehicle,  and  tracked  vehicle.  This 
is  a  large  subsection  of  the  total,  with  2,131  detections.  The  predetermined  distance  that  is  acceptable  for  all  vehicle 
detections  is  350  meters  based  on  the  BAIS  manual. 


In  Figure  21  you  will  see  the  baseline  case  and  the  filtering  cases  where  the  detection  threshold  was  two  and  three 
detections.  Again,  the  baseline  for  all  vehicles  has  a  high  number  of false-positives  with  88.5%.  Again  we  see  that  the 
filtering  reduces  the  number  of false-positives,  but  creates  some  fals e-negatives.  In  Figure  22  you  see  the  baseline 
repeated  and  the  filtering  cases  where  the  detection  threshold  was  four,  five,  six,  and  seven. 

Figures  23  and  24  are  charts  for  detection  thresholds  two  and  three.  Both  charts  focus  in  on  the  time  period  that  causes 
the  number  of false-positives  to  drop  close  to  zero.  The  main  thing  to  notice  from  these  charts  is  the  same  as  for  the 
previous  section  for  Unknown  Detections.  For  the  higher  thresholds,  a  relatively  small  change  in  acceptable  time  has 
little  to  no  effect  on  the  number  of false-positives.  Also,  after  a  threshold  of  three,  there  is  not  much  improvement  in 
results. 
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Figure  21:  All  Vehicle  Detections  with  Threshold  at  Two  and  Three 
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Figure  22:  All  Vehicle  Detections  with  Threshold  at  Four  through  Seven 
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Figure  23:  All  Vehicle  Detections  with 
Threshold  at  Two 
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Figure  24:  All  Vehicle  Detections  with 
Threshold  at  Three 


4.  CONCLUSION 

This  analysis  shows  that  for  all  cases,  the  baseline  has  a  very  high  number  of false-positives,  and  that  utilizing  a  filter 
will  reduce  the  false-positives  substantially  while  creating  some  fals  e-negatives.  However,  it  is  quite  apparent  that  the 
use  of  a  filter  greatly  increases  the  ratio  of  true  results  to  false,  which  causes  the  sensor  information  to  be  much  more 
reliable  and  useful.  The  best  filtering  methods  are  those  which  maximize  the  ratio  of  true  results  to  false.  It  is  clear  from 
the  graphs  shown  that  requiring  many  detections  in  a  small  amount  of  time  provides  the  best  true/false  ratio.  However, 
expecting  too  many  detections  in  too  small  of  a  time  will  degrade  results  due  to  limitations  of  the  BAIS  system.  A 
proper  balance  is  required. 

The  results  of  this  analysis  make  intuitive  sense  because  any  real  targets,  personnel  or  vehicles,  cannot  make  it  through 
the  coverage  area  of  the  BAIS,  as  much  as  350  meters  in  diameter  depending  on  soil  conditions,  during  a  single  10 
second  time  window,  thereby  increasing  the  likelihood  of  multiple  detections  on  a  single  incursion.  It  is  also  reasonable 
to  think  that  the  detecting  sensors  would  continue  alarming  at  the  maximum  rate  as  the  threat  in  the  area  persisted. 
Filtering  out  these  single  detection  events  removes  much  of  the  noise  in  the  system  from  such  sources  as  aircraft,  distant 
highway  traffic,  inclement  weather,  and  other  transient  sounds. 
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The  data  used  for  this  analysis  is  not  perfect,  though  it’s  obvious  that  a  truly  perfect  scenario  does  not  exist  in  real  life. 
However,  some  environmental  factors  definitely  did  come  into  play,  which  could  not  be  accounted  for  in  the  collected 
data.  These  factors  included:  traffic  on  the  nearby  highway,  traffic  from  site  personnel,  and  traffic  from  the  unmanned 
vehicles  also  being  used  in  the  experiment.  Despite  these  interferences,  this  analysis  should  be  considered  a  first  step  in 
properly  filtering  BAIS  sensor  detections,  when  deployed  in  high  traffic  areas. 
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